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DETAILED ACTION 

1 . This action is responsive to communications: amendment with the re-filed 
declaration and affidavit under 37 CFR 1.131 filed 6/2/05 to the application filed on 
6/28/00. 

2. Claims 1-27 are pending in the case. Claims 19-21 have been withdrawn from 
consideration. Claims 1,6, 10, 15, 19, 22, 25 are independent claims. 

Claim Rejections - 35 USC § 102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351(a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

4. Claims 1-18, 22-27 remain rejected under 35 U.S.C. 102(e) as being anticipated 
by Wesemann et al. (US Pat No. 6,349,132 B1 , 2/19/02, filed 12/16/99). 
Regarding independent claim 1 , Wesemann discloses: 

- identifying a Visual Browser and a Voice Browser, which are each implemented 
as functionally independent software components (figure 3, col 6, lines 18-36: 
accessing the HTML document by the Visual Browser and the Voice Browser 
shows that the two browsers are identified to provide two different interfaces by 
different softwares with different functions for presenting the HTML document) 
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- retrieving a network-based document formatted for in the Visual Browser (figures 
3-5; col 3, line 33 to col 4, line 6) 

- identifying in the retrieved document a reference to the Voice Browser, said 
reference specifying electronic content formatted for audible presentation in the 
Voice Browser (figures 3-4; col 5, lines 12-38) 

- transmitting said reference to the Voice Browser (figure 3: transmitting HTML 
documents in the HTML source #310 to the Voice Browser #340; col 5, lines 27- 
38: the system identifies the links in a document and provides users with an 
audio representation of the document text) 

- the Voice Browser retrieving said specific electronic content and audibly 
presenting said electronic content in the Voice Browser (figures 3-5: the 
document is retrieved and played audio of text after the text to speech 
translating; col 5, lines 27-38) 

- the Visual Browser visually presenting said network-based document 
concurrently with said audible presentation (col 6, lines 18-36: "Voice Browser 
#340 could be used in conjunction with Visual Browser #320 to provide 
simultaneously visual and audio interfaces..."; col 2, lines 20-37) 

- wherein the step of concurrently presenting results in a multi-modal presentation 
of the retrieved network-based document, the multi-modal presentation having a 
visual modality and an audible modality, functions for the visual modality being 
provided by the Visual Browser and functions for the audible modality being 
provided by the Voice Browser (col 6, lines 18-36: providing simultaneous visual 
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and audio interfaces on the monitor or on the telephone for presenting a HTML 
document shows a multi-modal presentation where the Visual Browser with 
visual functions provides visual presentation and the Voice Browser with audio 
functions provides audible presentation) 

Regarding claim 2, which is dependent on claim 1 , Wesemann discloses that the 
network-based document formatted for visual presentation in the Visual Browser is an 
HTML formatted document and the Visual Browser is a Web Browser (figure 3: the 
network document is a HTML document on the Internet so the visual browser #320 is a 
Web Browser; col 2, lines 20-52). 

Regarding claim 3, which is dependent on claim 1, Wesemann discloses that said 
electronic content formatted for audible presentation in the Voice Browser is VoiceXML 
formatted electronic content (col 5, lines 20-53). 

Regarding claim 4, which is dependent on claim 2, Wesemann discloses that said 
reference is a coordination markup attribute (col 2, lines 20-30 and col 8, lines 24-36: 
elements in the tags of a markup document include a markup attribute). 

Regarding claim 5, which is dependent on claim 1, Wesemann discloses that said 
reference specifies a network-based document containing said electronic document 
formatted for audible presentation in the Voice Browser (col 5, lines 27-38) whereby the 
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Voice Browser can retrieve said specified network-based document through the 
computer communications network and audibly present said electronic content 
contained in said specified network-based document, while the Visual Browser can 
visually present said network-based document formatted for visual presentation (col 6, 
lines 18-36; col 2, lines 33-37). 

Regarding independent claim 6 and its dependent claims 7-9, the claims are merely the 
opposite way to retrieve and present a network-based electronic document as in claims 
1-3, and thus are rejected as in claims 1-3. Wesemann discloses accessing network- 
based electronic content in both a Voice Browser and a Visual Browser where the web 
document is displayed and presented simultaneously in the Voice Browser and the 
Visual Browser Wiseman, therefore, inherently discloses the both ways of presentation 
a web document in an audio interface and a visual interface (col 3, line 33 to col 4, line 
6). Wesemen further discloses: 

- the Visual Browser lacks voice browsing capabilities and the Voice Browser lacks 
visual browsing capabilities (figure 3, col 6, lines 18-30: Visual Browser provides 
a visual presentation, and thus no voice browsing capabilities, and Voice 
Browser provides a voice presentation, and thus no visual presentation) 

- presenting the network-based document by the Voice Browser concurrently with 
the visual presentation in a coordinated and complementary fashion such that 
function of visual browsing provided by the Visual Browser are synchronized with 
functions of voice browsing provided by the Voice Browser (col 6, lines 30-36: the 
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fact that the Visual Browser and the Voice Browser could be used simultaneously 
through the telephone shows that visual functions provided by the Visual Browser 
are synchronized with audio functions provided by the Voice Browser since both 
visual interface and audio interface presented at the same time) 

Claims 10-14 are for a machine readable storage of method claims 1-5, and are 
rejected under the same rationale. 

Claims 15-18 are for a machine readable storage of method claims 6-9, and are 
rejected under the same rationale. 

Regarding independent claim 22, Wesemann discloses: 

- identifying a Visual Browser and a Voice Browser, which are each implemented 
as functionally independent software components (figure 3, col 6, lines 18-36: 
accessing the HTML document by the Visual Browser and the Voice Browser 
shows that the two browsers are identified to provide two different interfaces by 
different softwares with different functions for presenting the HTML document) 

- said Visual Browser lacking voice browsing capabilities and said Voice Browser 
lacking visual browsing capabilities (figure 3, col 6, lines 18-30: Visual Browser 
provides a visual presentation, and thus no voice browsing capabilities, and 
Voice Browser provides a voice presentation, and thus no visual presentation) 
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- incorporating visual content in a network-based document (figures 7-8; col 2, 
lines 38-52; col 3, lines 33-47) 

- formatting said network-based document for visual presentation in a Visual 
Browser (col 3, lines 23-47) 

- inserting at least one markup tag in said network-based document, said markup 
tag containing a coordination markup attribute specifying a network-based 
document formatted for audible presentation in a Voice Browser (col 2, lines 20- 
30, col 8, line 24-36 and col 5, lines 1 1-37: the links in the HTML document can 
be identified for playing the document text in the audio interface where the links 
are implemented by markup tags indicates that said markup tags can be included 
to the HTML codes for specifying the audio presentation of the web document 
which can be presented simultaneously with the visual presentation (col 6, lines 
28-38)) 

- whereby the Visual Browser when rendering said network-based document 
formatted for visual display can identify said coordination markup attribute in said 
markup tag and can transmit a reference to said specified network-based 
document and audibly present said specified network-based document 
concurrently with the visual presentation of said network-based document 
formatted for visual presentation in the Visual Browser (col 3, line 48 to col 4, line 
6 and col 6, lines 18-36) 

- presenting the network-based document by the Voice Browser concurrently with 
the visual presentation in a coordinated and complementary fashion such that 
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function of visual browsing provided by the Visual Browser are synchronized with 
functions of voice browsing provided by the Voice Browser (col 6, lines 30-36: the 
fact that the Visual Browser and the Voice Browser could be used simultaneously 
through the telephone shows that functions provided by the Visual Browser are 
synchronized w ith functions provided by the Voice Browser since both visual 
interface and audio interface presented at the same time ) 

Regarding claim 23, which is dependent on claim 22, Wesemann discloses that said 
network-based document formatted for visual presentation in a Visual Browser is an 
HTML formatted document and said Visual Browser is a Web Browser (figure 3: the 
network document is a HTML document on the Internet so the visual browser #320 is a 
Web Browser; col 2, lines 20-52). 

Regarding claim 24, which is dependent on claim 22, Wesemann discloses that said 
network-based document formatted for audible presentation in a Voice Browser is a 
VoiceXML formatted document and said Voice Browser is a VoiceXML Browser (col 2, 
lines 20-53, figures 1 , 3, col 6, lines 18-36). 

Regarding independent claim 25, Wesemann discloses: 

- identifying a Visual Browser and a Voice Browser, which are each implemented 
as functionally independent software components (figure 3, col 6, lines 18-36: 
accessing the HTML document by the Visual Browser and the Voice Browser 
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shows that the two browsers are identified to provide two different interfaces with 
different softwares for presenting the HTML document) 

- incorporating voice content in a network-based document (col 5, lines 1 1-38) 

- formatting said network-based document for audible presentation in a Voice 
Browser (col 6, lines 18-63) 

- inserting at least one markup tag in said network-based document, said markup 
tag containing a coordination markup attribute specifying a network-based 
document formatted for visual presentation in a Visual Browser (col 7, lines 10-53 
and figures 6-9: the displayed web document as seen is encoded by the markup 
tags containing the markup attribute for a visual presentation of the web 
document; col 8, lines 23-36) wherein said markup tag is used to synchronize the 
visual presentation of content from the network-based document within the Visual 
Browser and the audible presentation of content from the network-based 
document within the Voice Browser, and wherein said visual presentation and 
said audible presentation occurs in a concurrent, coordinated, and 
complementary fashion that results in a multi-modal presentation of content from 
the network-based document (col 5, lines 28-38: the links identified for presenting 
in audio presentation where the links are implemented by the markup tags and 
where the audio presentation can be performed simultaneously with the visual 
presentation inherently shows that the markup tags are used for synchronizing 
the visual presentation and the audio presentation) 
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Regarding claim 26, which is dependent on claim 25, Wesemann discloses that said 
network-based document formatted for visual presentation in a Visual Browser is an 
HTML formatted document and said Visual Browser is a Web Browser (figure 3: the 
network document is a HTML document on the Internet so the visual browser #320 is a 
Web Browser; col 2, lines 20-52). 

Regarding claim 27, which is dependent on claim 25, Wesemann discloses that said 
network-based document formatted for audible presentation in a Voice Browser is a 
VoiceXML formatted document and said Voice Browser is a VoiceXML Browser (col 2, 
lines 20-53 and col 10, lines 21-63). 

Response to Amendment 

5. The affidavit filed and the declaration re-filed 6/2/05 under 37 CFR 1.131 have 
been considered but remains ineffective to overcome the Wesemann reference 
(6,349,132). 

The evidence submitted on 6/2/05 that includes the claim mapping and the letter from 
IBM dated November 29, 1999, letter from Akerman to IBM dated December 6, 1999, 
and the redacted docketing notes of 3/29/00, 4/29/00, 5/29/00, and 6/28/00 seems to be 
sufficient to prove conception and diligence. However, the submitted evidence is not 
filed in a proper format. It is requested that Applicant either address all of the evidence 
in the declaration or incorporated by reference thereto instead of mentioning the 
evidence in the remarks. 
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Response to Arguments 

6. Applicant's arguments filed 6/2/05 have been fully considered but they are not 
persuasive. 

Applicants argue that Weseman's teachings of establishing a hierarchical index lacks 
teachings and mechanisms necessary for integrating or synchronizing the visual and 
audio interfaces of a Voice Browser and Visual Browser together (Remarks, page 14). 
Examiner respectfully disagrees. 

Wesemann teaches presenting the visual interface by the Visual Browser and the audio 
interface by the Voice Browser simultaneously (col 6, lines 30-36). This feature shows 
that the visual interface and the audio interface are synchronized to be presented at the 
same time. 

Applicants argue that Wesemenn does not teach coordination markup tags, which could 
be used to synchronize a Voice Browser and a Visual Browser since the category 
attributes used to map page content would not provide a means to synchronize and 
coordinate audio/visual content between the Visual Browser and Voice Browser as 
claimed (Remarks, page 14). 
Examiner respectfully disagrees. 

Wesemann discloses identifying the number of links and provides the user with an 
audio representation of the document text (col 5, lines 28-38). Since the links are 
implemented by markup tags, and since these tags are created for presenting the 
document text in the audio representation which can be simultaneous presenting with 
the visual presentation of the document (col 6, lines 30-36), the identified links in 
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Wesemann inherently show coordination markup tags used to synchronize the Voice 
Browser and the Visual Browser. 



Conclusion 

7. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 

§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

8. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Kuiken et al. (US Pat No. 6,922,733 B1, 7/26/05, filed 6/30/99). 
Miyashita et al. (US Pat No. 6,057,856, 5/2/00, filed 9/16/97). 
Brocious et al. (US Pat No. 6,745,163 B1 , 6/1/04, filed 9/27/00). 
Wynblatt et al. (US Pat No. 6,018,710, 1/25/00, filed 12/13/96). 
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Barbara et al., The AudioWeb, ACM 1997, pages 97-104. 

9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Cong-Lac Huynh whose telephone number is 571-272- 
4125. The examiner can normally be reached on Mon-Fri (8:30-6:00). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Stephen Hong can be reached on 571-272-4124. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-4125. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 




Cong-Lac Huynh 
Examiner 
Art Unit 2178 
08/17/05 



